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ABSTRACT 



One of a series of booklets on disability research, this 
paper is intended as an introduction to the role of evaluation in the 
utilization process. Its purpose is to help disability researchers grasp the 
importance of incorporating a focus on assessing use into plans for 
disseminating research outcomes. The paper begins by examining basic 
evaluation principles as they relate to dissemination and utilization, 
including: (1) evaluation is more than a compliance activity, its goal is 

improvement; (2) evaluation is not research; (3) effective evaluation must 
meet basic criteria relating to utility, feasibility, propriety, and 
accuracy; (4) there is no single, best way to do an evaluation; and (5) there 
are multiple dimensions to measuring use. The paper then describes the major 
steps involved in evaluation planning, including: determine who the intended 
users of the evaluation are, identify the purpose of the evaluation, specify 
the questions that the evaluation needs to answer, specify evaluation 
criteria, and identify evaluation methods. The paper concludes with a list of 
eight "use-deadly" sins that researchers should avoid. (CR) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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For the grantees of the National Institute on 
Disability and Rehabilitation Research (NIDRR), 
the term utilization activity is defined as: | 

I 

In carrying out a utilization activity under 
this program, a grantee must relate 
research findings to practical applica- 
tions in planning, policy making, pro- 
gram administration, and delivery of 
services to individuals with disabilities. 

i 

(Authority Section 202, 29 U.S. C. 761a; i 

Federal Register, 2/6/97. pp. 5711-5721.) j 




The importance of addressing utilization. Many disability 
researchers, when they begin planning for dissemination 
and utilization (D&U), think primarily about the “D.” 
Dissemination is the important item on most people’s 
agendas: how to get research results to intended audiences 
in the most effective, cost-efficient manner. If someone 
pauses to consider utilization , it’s generally to wonder 
why three words are employed when, it seems, one would 
suffice. The phrase dissemination and utilization seems 
too big a mouthful, literally and conceptually. 

But utilization is a critical element in increasing the 
effective reach of disability research outcomes. Focusing 
only on the “D” in D&U is like dialing nine numbers of 
a ten-digit telephone number: You may be 90 percent 
finished, but unless you dial that last digit, you’ll never 
make the correct intended connection. 

What do dissemination and utilization address? 

Where does one kind of activity end and the other begin? 
There is no single, clear line of demarcation. In fact, if 
you’re familiar with the literature (see NCDDR, 1996; 
NCDDR, 1999), you probably know that some definitions 
of dissemination go so far as to encompass use. Generally, 
though, dissemination has been unable to break free of 
its roots — its Latin roots, that is, its literal reference to 
scattering seed. People associate dissemination with 
spreading the word; how ideas and information get 
used seems another issue altogether. 
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So it’s helpful to pull the two ideas apart, to assure that 
each can be addressed explicitly. WeVe found the following 
to be a useful way of thinking about D&U: 

• Dissemination speaks primarily to the process of 
knowledge transfer — the who, what, when and how 
of moving ideas and information from the source(s) to 
intended recipient(s). 

• Utilization speaks primarily to purpose and to impact — 
why you want people to get the research outcomes 
you’re putting forth, what use you want people to 
make of the ideas, information, or products, and 

how people are actually using them. 



Evaluation as key to improving utilization. Central to the 
utilization portion of D&U is evaluation. To understand 
how effectively you are reaching your audiences — 
not merely in terms of whether audience members hear 
a Public Service Announcement about employment of 
people with disabilities, for example, but in terms of how 
listeners do or do not apply the information they hear — 
you must have some means of evaluating use. 

This paper is intended as an introduction to the role of 
evaluation in the utilization process. Its purpose is to help 
disability researchers grasp the importance of incorporating 
a focus on assessing use into plans for disseminating 
research outcomes. The paper examines basic evaluation 
principles as they relate to dissemination and utilization, 
and describes the major steps involved in evaluation 
planning. Future publications in this series will examine 
specific evaluation strategies that may be particularly 
useful to disability researchers. 
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Evaluation is too rarely thought of as a tool that is 
substantively important to a project’s implementation. 

If you’re like many people involved in research and 
development (R&D), you probably have little to do with 
the evaluation of your programmatic activities, either at 
the planning or the implementation stages. When preparing 
a proposal or plan of work, many people either bring 
in an evaluation specialist to draft the evaluation plan, 
or recycle language and procedures used in previous 
projects. Proposal writers are often careful to include 
both “formative” and “summative” activities, and to identify 
indicators if required, then it’s on to the “real” work of the 
project. Implementing the evaluation plan becomes the job 
of another department, or a staff member or consultant 
hired specifically for the task. Because the R&D program 
staff have had little or nothing to do with designing the 
evaluation, their attention to the results is based mostly 
on a desire for the project to get a “clean bill of health.” 
That is not the approach to evaluation we’re discussing 
here. The following are some basic ideas that may help 
bring evaluation into better focus as a practical tool that 
can be of real service in your work. 
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Evaluation is more than a compliance activity. Everything, 
including evaluation, comes back to the question of utility. 
According to Michael Quinn Patton, on whose book, 
Utilization-Focused Evaluation (Edition 3, 1997), this paper 
is based, the first thing to consider about evaluation is its 
use. The purpose of evaluation is not merely to comply 
with funding, guidelines or to lend an air of legitimacy to 
a research and development effort. Rather, the goal is 
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improvement. If your goal is to do the best possible job 
of developing and/or disseminating disability research 
outcomes that people actually use, then your evaluation 
activities should yield information that you and your staff 
can use. But it’s up to you to make evaluation useful, by 
getting involved in both its design and its implementation 



Evaluation is not research. Often people resist thinking 
about evaluation because they conceive of it as a 
necessarily elaborate, highly technical research activity. 
Although there are significant correspondences between 
the two, evaluation as we are discussing it does not 
necessarily require a research methodology in order 
to yield useful results. As Patton (p. 24) notes, 

Program evaluation differs fundamentally from 
research in the purpose of data collection and 
standards for judging quality. . . Research aims to 
produce knowledge and truth. Useful evaluation 
supports action. 

One important commonality that evaluation does share 
with research, however, is that, like research, evaluation 
must be grounded in reliable data. In some cases, you may 
use research methods to generate that data; or you may use 
“management information system data, program monitoring 
statistics, or other forms of systematic information that are 
not research-oriented” (Patton, p. 24). 
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Effective evaluation must meet basic criteria, if evaluation 
activities are to be of help in improving D&U, they should 
meet four basic criteria, or standards of quality. These 
criteria were developed in 1980 by a national Joint 
Committee on Standards, representing a number of 
professional organizations, and were restated in 1994. 

The four basic criteria are: 

Utility 

Utility involves taking steps to ensure that “an evaluation 
will serve the practical information needs of intended 
users.” As committee chair Daniel Stufflebeam observed, 

“an evaluation should not be done at all if there is no 
prospect for its being useful to some audience.” 

Feasibility 

To be judged as feasible, an evaluation must be “realistic, 
prudent, diplomatic, and frugal.” SRifflebeam suggests an 
evaluation should not be conducted “if it is not feasible 
to conduct it in political terms, or practicality terms, or 
cost-effectiveness terms.” 

Propriety 

Propriety involves ensuring that “an evaluation will be 
conducted legally, ethically, and with due regard for 
the welfare” of both the individuals participating in the 
evaluation and those who are affected by its results. 

Accuracy 

To be accurate, an evaluation must “reveal and convey 
technically adequate information about the feaaires that 
determine worth or merit” of what’s being evaluated. 

This criterion addresses the methodological appropriateness 
of both the evaluation plan and its implementation 
(Patton, pp. 16-17). 
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There is no single, best way to “do” an evaluation. There are 
many approaches to evaluation that can meet the four basic 
criteria listed above. “The notion that there is one right way 
to do things dies hard.” Patton concludes: 

Eveiy evaluation situation is unique. A successful 
evaluation (one that is useful, practical, ethical, and 
accurate) emerges from the special characteristics 
and conditions of a particular situation — a mixture 
of people, politics, history, context, resources, 
constraints, values, needs, interests, and chance. 

(p. 126) 

According to evaluation experts including Patton, designing 
the “right” evaluation for any specific program “requires 
interaction, negotiation, and situational analysis.” It also 
involves keeping a clear eye on the purpose of the 
evaluation activity, which is to yield useful results. Utility 
should never have to play second fiddle to methodology. 



There are multiple dimensions to measuring use. The role 
of evaluation in relation to the D&U process is to help 
disability researchers understand how much and how 
effectively consumers are using the research outcomes that 
have been disseminated. Utilization of research outcomes, 
then, can be considered in terms of two basic dimensions. 

When researchers think about evaluating consumers 7 
use, they generally think of only one of these dimensions: 
the level or extent of use. Patton describes this dimension 
as the “use - nonuse continuum.” Evaluation along this 
continuum involves measuring the degree or magnitude 
of use among intended audiences. 

There is a second, equally important, dimension, 
however: the use - misuse continuum. In most instances, 
it's important to assess the manner as well as the degree of 
use of disability research outcomes, asking and obtaining 
answers to the question: How — and how effectively — 
are consumers using research outcomes? 
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One important caution about evaluation along the 
use - misuse continuum: The common tendency is to focus 
on evaluating use in terms of the researchers’ intentions , 
that is, assessing the extent to which consumers actually 
use research outcomes in the way that disability researchers 
intended. But intended use and effective use are not always 
synonymous. Sometimes consumers make adaptations 
or find effective uses never considered by researchers; 
sometimes cultural differences between researchers and 
consumers shift the assumptions as to what constitutes 
effective use. 

You may want to know whether consumers are using 
research outcomes as you intended; certainly that’s a 
legitimate and important function of evaluation. It’s also 
important, though, to include in your data collection plan 
ways of looking beyond your specific intentions, and 
to be careful in the assumptions you make about use 
versus misuse . 
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Steps in the Evaluation 
Planning Process 

As is true for dissemination, evaluation planning needs 
to begin in the earliest stages of a project. Planning 
evaluation activities that will yield useful results, however, 
is likely to be quite different from the pro forma or 
compliance-oriented planning typical of many research 
and development (R&D) projects. The following steps are 
key to planning an effective, utilization-focused evaluation. 

Determine who the intended users of the evaluation are. 

The first step is to identify the people who will use the 
evaluation results, so that evaluation questions and results 
can be tailored to their needs. Patton observes, “Clearly and 
explicitly identifying people who can benefit from an evalu- 
ation is so important that evaluators have adopted a special 
term for potential evaluation users: stakeholder f (p. 41). 

In evaluating the dissemination and utilization of 
disability research outcomes stakeholders may include 
many groups, including the grant-sponsored staff and other 
organization staff, managers, advisory boards, boards of 
directors, funding sources, consumers and their families, 
and direct service providers. It is difficult, and often 
impossible, to devise evaluation activities that address 
the diverse interests of all these stakeholders. 

That means that some process is necessary for 
narrowing the range of possible questions to focus 
the evaluation. In utilization-focused evaluation, this 
process begins by narrowing the list of potential 
stakeholders to a much shorter, more specific group 
of primary intended users. Their information needs, 
that is, their intended uses , focus the evaluation 
(Patton, p. 42). 
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The disability researchers involved in developing and 
disseminating research outcomes are, of course, an impor- 
tant stakeholder group whose information needs should be 
addressed in any evaluation activity. But inclusion is not 
enough. As studies conducted by Patton and others show, it 
is people’s interest in, and commitment to, using evaluation 
results that determine whether those results actually 
support program improvement. Patton describes this as 
“the personal factor,” and offers the following definition: 

The personal factor is the presence of an identifi- . 
able individual or- group of people who personally 
care about the evaluation and the findings it 
generates. Where such a person or group was 
present, evaluations were used; where the personal 
factor was absent, there was a correspondingly 
marked absence of evaluation impact (p. 44). 

It is essential, then, for disability researchers and related 
program staff to recognize the ways in which evaluation 
results can strengthen their work, and to think of evaluation 
as an integral part of their programmatic work. 



2 Identify the purpose Of the evaluation. Three primary purposes 
for evaluation include: “rendering judgments, facilitating 
improvements, and/or generating knowledge” (Patton, p. 
65). For disability researchers, the first two purposes will 
generally be of greatest concern. Most evaluation efforts 
will be concerned with generating data that demonstrate 
the project’s value to funding sources and institutional 
leaders and that help the project and its successors 
work more effectively. 

While it is possible for a single evaluation to address 
more than one, even all three, of these basic purposes, 
there are hazards in doing so. 

One [purpose] is likely to become the dominant 
motif and prevail as the primary purpose informing 
design decisions and priority uses. . . I also find that 
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confusion among these quite different purposes, 
or failure to prioritize them, is often the source of 
problems and misunderstandings along the way 
and can become disastrous at the end when it 
turns out that different intended users had different 
expectations and priorities (Patton, p. 67). 

The fundamental differences between evaluation results 
that are used to render judgments about program merits 
and those used for program improvement may be 
characterized as: 

Judgment-oriented evaluation requires preordinate, 
explicit criteria and values that form the basis for 
judgment. Improvement-oriented approaches tend 
to be more open ended, gathering varieties of 
data about strengths and weaknesses with the 
expectation that both will be found and each can 
be used to inform an ongoing cycle of reflection 
and innovation (Patton, p. 68). 

Given that, in most cases, any evaluation of your 
efforts will need to address both of these purposes, it is 
important to design an evaluation that gets at both kinds 
of information. And just as questions of utility should not 
play second fiddle to a rigid methodological perspective 
(as we noted earlier), they shouldn’t be subordinated to 
questions of judgment, either. 
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Specify the questions that the evaluation needs to answer. 

This step is really a further elaboration of the evaluation’s 
purpose. Most people are well acquainted with judgment- 
related questions, which shape most compliance-oriented 
evaluations. In addition, Patton (p. 68) identifies a number 
of questions that are relevant to improvement-oriented 
evaluation. In the list on page 11, some questions 
have been adapted to more specifically address 
disability research: 

15 



National Center for the Dissemination of Disability Research 



Steps in the Evaluation Planning Process 



Questions that the evaluation needs to answer. 

• What are the research project’s strengths 
and weaknesses? 

• To what extent are intended audiences making 
use of disability research outcomes? 

• Which types of consumers or stakeholders are 
using your disability research outcomes and 
which types aren’t? 

• Which types of dissemination media have been 
most effective in reaching targeted consumers? 
Which have been least effective? 

• How are consumers using disability research 
outcomes? How closely do those uses 
correspond with project staffs identified uses? 

• What kinds of implementation problems have 
emerged and how are they being addressed? 

• What’s happening that wasn’t expected? 

• How are research project staff and consumers 
interacting? 

• What are staff and consumer perceptions of 
the program? What do they like? dislike? want 
to change? 

• What are perceptions of the research project’s 
culture and climate? 

• How are funds being used compared to initial 
budgetary expectations? 

• How is the research project’s external 
environment affecting internal operations? 

• Where can increased or improved efficiencies 
be realized? 

• What new ideas are emerging that can be 
tried out and tested? . 

IS 
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Specify evaluation criteria. As noted earlier, the importance 
of delineating evaluation criteria varies according to 
the evaluation’s purpose. Patton emphasizes that, “in 
judgment-oriented evaluations, specifying the criteria 
for judgment is central and critical” (p. 66). Different 
stakeholders may have far different perceptions as to 
what is important in determining a research project’s 
success. For example, a legislator who helped to authorize 
a funding initiative may be most concerned about the 
number of consumers reached by a project, while project 
staff may be much more concerned about the extent 
of behavior change among a much smaller, targeted 
beneficiary group. It’s important, then, to clarify 
expectations at the planning stage, so that evaluation 
methods will yield the kinds of information you need. 

Evaluation criteria are often derived from the goals and 
objectives outlined in a grant proposal or other plan of 
work. As Patton points out, however, there are often big 
problems with this approach. Such goals tend to be stated 
in terms of research project activities or services rather than 
in terms of consumer outcomes. Patton itemizes a number 
of examples of these problematic goals/objectives, such as: 



To develop needed services for chronically 
chemically dependent clients. 

Develop a supportive, family-centered, empowering, 
capacity-building intervention system for families 
and children (p. 155). 



What’s needed instead is “a clear statement of 
the targeted change in circumstances, status, level of 
functioning, behavior, attitude, knowledge, or skills” 
among intended users (Patton, p. 159). 

Because of the difficulty in collecting data that fully 
capture the kinds of changes listed in the preceding 
paragraph, it’s usually necessary to come up with some 
indicators that reflect the desired changes you want to 
assess. Patton emphasizes the need to remember that 
O outcomes and': indicators are not synonymous. 
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Some kind of indicator is necessary, however, to 
measure degree of outcome attainment. The key is to 
make sure that the indicator is a reasonable , use- 
ful, and meaningful measure of the intended client 
outcome (Patton, pp. l60-l6l). 

As stated earlier, improvement-oriented evaluation 
is much more open-ended than is judgment-oriented 
evaluation. Patton notes that there are alternatives to 
“goals-based evaluation” (p. 177). One alternative, labeled 
“goal-free evaluation” by evaluation expert Michael Scriven 
(cited in Patton, p. 181), involves “gathering data on a 
broad array of actual effects and evaluating the importance 
of these effects in meeting [the] demonstrated needs” 
of intended audiences. Another alternative that Patton 
recommends, and one that addresses the improvement- 
related information needs of disability researchers, is to 
focus on the kinds of questions that the users of evaluation 
results want answered, questions such as those listed in the 
preceding section. Rather than specific criteria for success, 
the questions serve to shape the evaluation methodology 
and data analysis. 

Even if you focus your improvement-oriented evaluation 
activities on such questions, however, it will likely be 
necessary to generate some explicit statements related to 
your programmatic intentions. For example, in order to 
assess the extent to which consumers are using research 
outcomes as researchers intended, it will be necessary to 
explicitly describe those intended uses and what constitutes 
“use” on the part of the consumer. 



Identify evaluation methods. One of the hazards in trying 
to shift to utilization-focused evaluation is that, in the 
planning process, project staff may work their way through 
the preceding four steps and then, at step five, aim the 
planning over to an evaluation expert. These evaluation 
specialists, in turn, often simply revert back to old, 
familiar but non-utilitarian evaluation methods. In 
contrast, Patton notes: 
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In utilization-focused evaluation, methods decisions, 
like decisions about focus and priority issues, 
are guided and informed by our evaluation goal: 
intended use [of the evaluation results] by intended 
users (p. 241). 

The best way to achieve this goal is to involve those 
intended users — meaning, among others, you and 
your research project staff — in making decisions about 
evaluation methods. What’s needed is an evaluator who 
can help guide you through this decision making process. 

The utilization-focused evaluator advises intended 
users about options; points out the consequences of 
various choices; offers creative possibilities; engages 
with users actively, reactively, and adaptively to 
consider alternatives; and facilitates their methods 
decisions. At the stage of choosing methods, the 
evaluator remains a technical adviser, consultant, 
and teacher. The primary intended users remain 
decision makers about the evaluation (Patton, 
p. 243). 

There are, of course, a number of important decisions to 
be made regarding the evaluation methodology; these may 
include the use of: 

• Mail questionnaires, telephone interviews, or personal 
face-to-face interviews? 

• Individual interviews or focus groups? 

• Even-numbered or odd-numbered scales on survey items? 

• Opinion, knowledge, and/or behavioral questions? 

• All closed questions or some open-ended? If some 
open-ended, how many? 

• Norm-referenced or criterion-referenced tests? 
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Development of our own instruments or adoption of 
measures already available? 
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• Experimental design, quasi-experimental design, or 
case studies? 

• Participant observation or spectator observation? 

• A few in-depth observations or many shorter 
observations? 

• Standardized or individualized protocols? 

• Fixed or emergent design? 

• Follow up after two weeks, three months, six months, 
or a year? 

• Follow up everyone or a sample? 

• What kind of sample: simple random, stratified, 
and/or purposeful? 

• What size sample? 

• Will interviewers have the same characteristics as 
program participants: gender? age? [disability?] race? 

• What comparisons to make: past performance? intended 
goals? hoped-for goals? other research project outcomes? 
(Patton, p. 247) 

The list of decisions can seem overwhelming, but with 
good technical assistance, the processing is not as daunting 
as it may seem. One helpful bit of information, is that, 
contrary to the beliefs of most non-researchers, “there 
are no universal and absolute standards for judging” the 
appropriateness of evaluation methods. Rather, the criteria 
for quality “are necessarily situational and context bound” 
(Patton, p. 249). 
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Adapting utilization-focused evaluation represents a 
significant departure from the more typical current practice 
in the field. The challenge to change is by no means trivial; 
it requires more work of project staff and — perhaps most 
difficult — a reorientation in thinking on the part of both 
evaluators and project staff. There are many factors that may 
tend to divert you, your staff, and the evaluators engaged in 
your project from a focus on utilization of evaluation results. 
Patton suggests eight such diversions, and calls them seven 
“use-deadly” sins (plus one). These include: 



1 . Making evaluators the primary decision makers and, 
therefore, the primary users. 

2 . Identifying vague, passive audiences as users [of 
evaluation results] instead of real people. 



3 . Targeting organizations as users (e.g. ‘the feds’) 

j instead of specific persons. 

I 



i 



i 

i 

! 



4 . Focusing on decisions instead of decision makers. 

5 . Assuming die evaluation’s “funding entity” is 
automatically the primary stakeholder. 

6. Waiting until the findings are in to identify intended 
users and intended uses. 



7 . Taking a stance of standing above the fray of 
people and politics. 




8 . Identifying primary intended users at the outset of 
the srticly, then ignoring them until the final report 
is ready (Patton, pp. 58, 60). 
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With a genuine commitment to project/practice 
improvement, however, it is more than possible to shift 
your evaluation plans to produce results you can actually 
use. Given the imperative to increase the use of disability 
research outcomes, we must all do no less. 
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admin@ncddr.org 

Write Us 
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211 East Seventh Street, Suite 400 
Austin, Texas 78701-3281 
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Visit Us 

In downtown Austin, Texas 4th floor, 

Southwest Tower, Brazos at 7th St. 

8 a.m.-noon and 1 p.m -5 p.m. C.T. 

Mon.-Fri. (except holidays) 
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Fax Us 

512-476-2286 
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® 




NOTICE 



Reproduction Basis 




This document is covered by a signed "Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 




EFF-089 (3/2000) 




